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“techniques  be  used  for  training  to  be  effective.  Several  dynamic 
skill  tasks  were  studied,  and  a  laboratory  analog  to  the  Air 
Intercept  Controller  task  was  developed  for  experimental  use.  The 
experiments  conducted  showed  that  intrusive  instruction  is  less 
effective  for  such  simulation  training  than  is  non-intrusive  in¬ 
struction.  Experiments  on  the  use  of  voice  input  and  voice  output 
devices  for  such  training  produced  mixed  results,  suggesting  that 
current  low-cost  technologies  for  voice  I/O  are  approaching  the 
acceptability  threshold  for  this  type  of  application. 
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ABSTRACT 


This  is  the  Pinal  Report  for  Contract  N00014-80-C-0164, 
covering  a  period  of  two  and  three-quarter  years,  from  February 
1980  through  Ootober  1982.  Research  was  performed  on  topics 
associated  with  computer  based  training  of  dynamic  skill  tasks. 
Training  for  such  tasks  is  to  be  distinguished  from  the  teaching  of 
fact  systems,  the  training  domain  most  heavily  studied  in  computer 
based  instruction  applications.  It  was  hypothesized  that  the 
attentional  demands  of  dynamic  skill  simulation  training  require 
that  special  instructional  techniques  be  used  for  training  to  be 
effective.  Several  dynamic  skill  tasks  were  studied,  and  a 
laboratory  analog  to  the  Air  Intercept  Controller  task  was 
developed  for  experimental  use.  The  experiments  conducted  showed 
that  intrusive  instruction  is  less  effective  for  such  simulation 
training  than  is  non-intruslve  instruction.  Experiments  on  the  use 
of  voice  input  and  voice  output  devices  for  such  training  produced 
mixed  results,  suggesting  that  ourrent  low-cost  technologies  for 
voice  I/O  are  approaching  the  acceptability  threshold  for  this  type 
of  application. 
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Its  Research  IsailfiS. 


Decreased  costs  and  increased  performance  for  dedicated 
microcomputer-based  simulation  training  systems  will  ensure  that 
these  systems  will  be  more  widely  implemented  than  in  the  past.  The 
prospect  of  such  widespread  implementation  motivates  the  study  of 
the  effective  use  of  such  systems  for  the  training  of  real-time, 
event-driven  skills.  The  objective  of  this  research  was  to 
develop  an  experimental  computer-based  training  system  and  to 
conduct  a  number  of  experiments  in  dynamic  skill  training  using  the 
system.  One  set  of  experiments  tested  different  approaches  to 
giving  the  student  Information  about  the  correctness  of  his 
performance  during  training.  A  second  set  of  experiments  focused 
on  the  problems  and  potentials  of  the  use  of  voice  input  and  voice 
output  technologies  in  computer  based  training. 

Simulators  have  long  been  known  to  provide  effective  practice. 
Transfer  to  actual  tasks  has  been  demonstrated  (Orlansky  and  Spring, 
1975).  We  believe  that  dynamic  skill  simulators  can  be  even  more 
effective  trainers  if  they  provide  instructional  functions  in 
addition  to  the  presentation  of  simulated  practice  episodes.  The 
computers  that  control  a  simulation  should  also  be  able  to  play  a 
role  in  guiding  the  training  process.  They  should  be  able  to 
interactively  tutor  the  student,  to  select  and  control  appropriate 
simulated  problems,  and  to  give  the  stude  t  useful  instructional 
feedback  during  training. 


At  present,  little  Is  known  about  what  techniques  are  likely 
to  prove  most  effective  In  training  dynamic  skills.  The  range  of 
techniques  tnat  could  be  applied  through  computer  controlled 
training  has  not  been  explored.  One  stimulus  for  this  research 
project  was  the  expectation  that  fact  system  Instruction  techniques 
taken  from  conventional  computer  assisted  instruction  (CAI)  would 
not  prove  to  be  an  effective  means  of  designing  computer  based 
dynamic  skill  training  systems. 

A  central  issue  of  concern  in  the  project  was  to  explore 
methods  for  mixing  Instruction  and  dynamic  skill  simulation 
practice.  In  conventional  CAI  for  fact  systems,  student  responses 
are  typically  required  only  after  the  system  has  finished  making 
some  presentation  to  the  student.  When  the  student  responds,  the 
system  evaluates  the  response  and  then  reacts  appropriately.  After 
making  a  response,  a  student’s  attention  is  typically  directed  to 
the  system,  in  expectation  of  evaluative  feedback  or  instruction 
relevant  to  the  response  just  made.  In  dynamic  skill  practice 
sessions,  in  contrast,  students  typically  expect  further 
simulation  responses  from  the  system  in  reaction  to  their  inputs. 
Their  attention  is  directed  to  the  simulated  events,  not  to 
tutorial  feedback.  How  can  such  instruction  be  effectively 
combined  with  simulation  practice?  The  first  series  of  experiments 
was  designed  to  address  this  issue. 
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Aa  Experimental  nvnanH »  Skill  Simulation  Training  System 


One  of  the  first  project  requirements  was  to  develop  a  dynamic 
skill  task  for  use  in  the  laboratory.  A  number  of  Navy  tasks  were 
studied  with  an  eye  to  producing  a  microcomputer  simulation  for 
training  research.  The  following  jobs  were  studied:  Ground 
Controlled  Approach  (GCA)  Controller,  Landing  Signal  Officer  (LSO), 
and  Air  Intercept  Controller  (AIC).  The  AIC  task  was  selected 
because  it  offered  the  opportunity  to  train  an  attention-demanding 
task  that  could  be  simulated  with  relatively  simple  and  inexpensive 
displays  and  controls. 

The  AIC  dynamic  skills  trainer  program  simulates  a 
videogame-like  task  similar  to  that  of  an  air  Intercept  controller. 
The  task  requires  that  the  student  observe  blips  representing 
aircraft  detected  by  radar,  and  then  use  a  Joystick  and  special 
function  keys  to  label  those  blips.  The  keyboard  is  also  used  to 
communicate  with  the  simulated  pilots  of  the  aircraft  to  get  fuel 
and  weapons  status  and  to  send  instructions  to  intercept  and  fire 
at  enemy  aircraft.  Students  are  trained  on  the  task  in  a  single 
training  session. 

A  student  training  episode  has  two  major  components.  The 
first  is  a  pre-training  session,  in  which  the  student  is  introduced 
to  the  task  and  to  the  activities  that  comprise  the  skill.  The 
second  is  a  practice  training  session,  in  which  the  student 
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receives  a  combination  of  tutorial  and  simulated  practice.  The 
tutorial  messages  do  not  follow  practice  problems,  but  are 
presented  during  them.  One  research  focus  was  how  to  mix  practice 
and  tutorial  instruction  effectively. 

In  the  pretraining  session  students  watch  a  computer-controlled 
videotape  demonstration  of  the  task  with  narrated  instructions.  Then 
brief  explanations  of  each  of  the  controls  available  to  the  student 
are  presented  in  text  format  on  the  simulated  control  console.  After 
reading  each  explanation,  the  student  is  required  to  briefly  interact 
with  a  simple  simulation  that  requires  the  exercise  of  the  function 
just  discussed. 

During  the  practice  training  session,  the  student  uses  the 
techniques  presented  in  the  pre- training  to  play  the  air  intercept 
simulation  game.  During  this  training  phase,  the  system  records 
student  actions,  the  occurence  of  simulated  events,  student  errors, 
and  tutorial  interactions  with  the  student. 

The  training  system  was  implemented  using  an  Apple  II  system 
with  64K  bytes  of  RAM,  a  hard  disc  drive,  a  real-time  clock 
interface,  a  joystick,  and  external  80-column  video  display  terminal, 
a  videocassette  controller,  and  interfaces  for  two  different  voice 
output  devices.  The  student  had  three  display  screens:  one  for  the 
videotaped  instructions,  an  80-column  text  display,  and  an  Apple 
Graphics  display  screen.  In  some  experiments,  the  student  wore  a 


headphone  that  was  connected  to  four  sound  sources:  the  videotape 
audio,  the  Apple's  speaker,  and  the  two  computer  voice  output 
devices. 

The  same  basic  hardware  configuration  and  core  training  program, 
described  in  technical  report  ONR-96,  were  used  to  Implement  a 
number  of  experiments. 
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Technical  Report  No.  ONR-96 

Allen  Munro,  Douglas  M.  Towne,  &  Michael  R.  Fehling,  An  Experimental 

System  for  Research  on  Dynamic  Skills  Training,  September  1 981 

Academic  research  on  computer  based  instruction  (CBI)  has  dealt 
largely  with  CBI  of  knowledge  systems,  coherent  bodies  of  essentially 
propositional  knowledge.  Little  research  has  been  performed  on 
techniques  for  effective  CBI  of  dynamic  skills,  those  amalgams  of 
perceptual,  motor,  and  decision-making  skills  that  are  required  by  many 
real-time  event-driven  tasks.  The  demands  of  dynamic  skill  training  on 
student  processing  resources  are  different  from  those  of  knowledge 
system  teaching.  These  differences  suggest  that  the  techniques  found 
to  be  effective  in  conventional  CBI  may  not  be  applicable  to  dynamic 
skill  training  CBI.  Two  classes  of  research  issues  to  be  explored  are 
techniques  for  presentation  of  simulation  practice  and  methods  for 
providing  effective  instructional  feedback. 

A  microcomputer-based  experimental  simulation  training  system  for 
research  on  dynamic  skill  training  is  described.  Experimental  subjects 
are  taught  to  perform  a  simulation  task  based  on  the  Job  of  an  air 
intercept  controller.  The  training  program  permits  controlled 
differences  in  instructional  treatment  for  different  groups  of 
students,  in  order  to  explore  empirical  issues  in  dynamic  skill 
training. 
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Intrusion  iQ  Dynamic  Skill  Instruction 


Dynamic  skill  training  Is  characterized  by  continuous 
involvment  of  both  the  computer  system  and  the  student.  There  Is  no 
clear  exchange  of  turns  between  the  computer  and  the  student  as  there 
Is  in  conventional  "fact  system"  computer-based  instruction.  The 
student  can  enter  a  response  at  the  same  time  that  the  simulation 
display  is  changed.  Because  the  student  is  always  busy  with  the 
task,  there  is  no  surrendering  of  the  student's  turn  in  order  for  the 
computer  to  instruct.  In  such  an  instructional  system,  how  can 
instruction  be  integrated  with  simulation  practice? 

Two  approaches  to  providing  informative  feedback  in  dynamic 
skill  training  were  compared  in  a  computer  based  training 
experiment.  The  first  approach  was  to  provide  corrective 
instruction  as  soon  as  the  student  made  a  mistake  during  simulated 
practice  training.  This  was  called  the  intrusive  instruction  mode. 
The  second  approach  was  to  signal  the  student  that  an  instructional 
message  was  available  whenever  he  or  she  made  a  mistake,  but  not  to 
present  the  instruction  until  the  student  requested  it.  This  was 
the  non-intrusive  instruction  mode. 

Two  groups  of  students  were  trained  to  perform  the  simulated  air 
intercept  control  task.  Each  group  of  students  received  the  same 
pre-training  and  worked  the  same  practice  problems.  One  group 
received  intrusive  and  the  other  non-intrusive  instruction.  The  non- 
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intrusive  group  made  significantly  fewer  errors  during  practice,  but 
time  spent  on  problems  was  not  significantly  different  for  the  two 
groups.  Analysis  of  errors  by  type  showed  that  both  important  and 
less  Important  aspects  of  performance  were  significantly  affected  by 
the  Intrusiveness  variable,  but  that  basic  motor  skills  did  not  seem 
to  be  affected  by  Intrusiveness.  Instead,  intrusive  instruction 
appears  to  reduce  cognitive  performance  on  the  task. 

These  results  can  be  interpreted  in  light  of  an  attentlonal 
demand  hypothesis.  In  a  complex  dynamic  skill  training  session,  most 
of  a  student *s  cognitive  processing  resources  are  likely  to  be 
allocated  to  attending  to  and  responding  to  the  task  itself.  If  the 
task  is  suddenly  intruded  upon  by  an  instructional  message,  the 
intrusion  will  demand  additional  processing  resources  to  perform 
the  attentlonal  shift.  This  surge  in  processing  resource  demand  is 
likely  to  Interfere  with  the  normal  learning  and  performance 
process.  If  the  disruption  occurs  at  a  point  in  the  task  when  a 
large  percentage  of  cognitive  resources  are  already  committed,  then 
either  attention  to  instruction  or  performance  on  the  task  is 
likely  to  suffer. 
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Technical  Report  No.  ONR-97 


Allen  Munro,  Michael  R.  Fehling,  Pierre  Blais,  A  Douglas  M.  Towns, 
Intrusive  and  Non- Intrusive  Instruction  in  Dynamic  Skill 
Training,  October,  1981. 


ABSTRACT 

A  distinction  is  drawn  between  computer  based  instruction  of 
knowledge  systems  and  computer  based  instruction  of  dvnami e  skills. 
There  is  reason  to  expect  that  the  findings  of  research  on  knowledge 
system  instruction  will  not  apply  universally  to  dynamic  skill 
instruction.  In  particular,  a  theory  of  cognitive  resource  demand 
suggests  that  the  principle  of  immediate  instructional  feedback  may  not 
apply  in  dynamic  skill  training.  Because  students  in  dynamic  skill 
training  are  often  heavily  loaded  with  processing  demands, 
instructional  feedback  must  be  postponed  until  the  students  have 
sufficient  free  resources  to  process  it.  This  hypothesis  was  tested  in 
an  experiment  in  computer  based  instruction.  One  group  of  students 
received  instructional  feedback  upon  request,  while  a  second  group 
received  feedback  under  program  control.  The  group  with  control  over 
feedback  made  significantly  fewer  errors  in  training  than  did  the  group 
that  did  not  control  timing  of  the  instructional  feedback  messages. 


Cobpu ter- baaed  Voice  Instruction  and  Intrusion 


One  concern  about  the  results  of  the  first  study  was  that  the 
task  was  so  difficult  that  students  processing  resources  were  heavily 
overloaded.  Would  the  intrusive  instruction  effect  hold  up  when  the 
task  was  less  demanding?  A  revised,  simplified  version  of  the  A1C 
task  was  developed  in  order  to  replicate  the  previous  findings  in  a 
simpler  task  environment. 

The  second  issue  addressed  in  the  new  study  was  the  consequences 
of  computer  generated  voice  output  in  simulation  and  instruction. 

The  experiment  addressed  the  question:  Are  currently  available  low 
cost  voice  output  devices  useful  for  instruction  in  dynamic  skill 
training?  Many  dynamic  skills  require  the  use  of  voice.  A  natural 
approach  to  computer  based  training  of  these  skills  is  to  make  use 
of  computer  generated  voice.  The  experiment  compared  instruction 
by  computer  generated  voice  with  text  instruction. 

Two  very  distinguishable  computer  generated  voice  output  devices 
were  used  in  the  voice  conditions  of  the  simulation  training.  A 
device  employing  a  pre-recorded  digital  representation  of  actual 
human  speech  was  used  to  simulate  the  vocal  responses  of  the  pilots 
of  the  controlled  aircraft.  This  device  produced  very  dear,  quite 
natural-sounding  speech.  A  text-to-speech  synthesis  device  was  used 
to  deliver  instructional  messages  to  students  in  the  voice  training 
group.  Students  in  the  text  group  read  the  same  messages.  The 
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speech  quell ty  of  the  text- to- speech  synthesiser  was  such  lower  then 
that  from  the  device  that  simulated  the  pilots'  volees.  During 
pre-training,  the  students  In  the  voloe  group  were  familiarised 
with  the  messages  by  listening  to  the  computer-generated  voloe 
while  reading  the  messages.  They  were  given  the  option  of 
repeating  each  message  until  they  understood  and  heard  them 
clearly. 

Students  in  the  Intrusive  instruction  groups  made  more  errors 
than  those  In  the  non-intrusive  Instruction  groups.  Analysis  of 
crucial  and  non-crucial  errors  revealed  that  there  was  a  significant 
difference  in  number  of  errors  only  for  non-crucial  errors.  This 
result  contrasts  with  that  of  the  previous  study,  in  which  cruoial 
errors  also  were  significantly  greater  for  the  intrusive  than  for  the 
non-intrusive  group.  It  is  likely  that  intrusive  group  students  were 
able  to  perform  as  well  as  non-intrusive  students  on  the  crucial  sub¬ 
tasks  because  the  revised  AIC  task  was  simpler  than  the  original 
task. 


Voice  instruction  resulted  in  significantly  more  errors  than  did 
text  instruction.  The  result  suggests  that  the  low-quality  voice 
output  equipment  used  in  this  experiment  is  not  appropriate  for 
dynamic  skill  training  tasks  such  as  the  AIC  task.  It  remains  to  be 
determined  whether  a  more  intelligible  voice  output  device  would 
result  in  performance  equal  to  or  superior  to  text  instruction 
performance. 
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Technical  Report  No.  ONR-99 


Allen  Munro,  James  Cody,  and  Douglas  M.  Towne,  Instruction  Mode 
Instruction  Instrusiveness  in  Dynamic  Skill  Training,  August 
1982. 


ABSTRACT 

Onllke  computer  based  instruction  of  knowledge  systems, 
instructional  feedback  for  dynamic  skill  training  has  been  found  to 
be  most  effective  when  the  student  chooses  when  and  if  feedback  is 
to  be  received  (Munro,  Fehllng,  Blais,  &  Towne,  1981).  Because 
students  in  dynamic  skill  training  are  often  heavily  loaded  with 
processing  demands,  instructional  feedback  must  be  postponed  until 
students  have  sufficient  free  resources  to  process  it.  The  present 
study  attempts  to  replicate  these  findings  using  a  simpler  task. 

The  second  factor  in  the  present  study  is  the  effectiveness  of 
computer  generated  voice  output  in  instruction  and  simulation  in 
dynamic  skill  training.  These  hypotheses  were  tested  in  an 
experiment  in  computer  basAd  instruction.  Both  the  intrusiveness 
and  delivery  mode  (text-voice)  factors  had  statistically 
significant  effects  on  student  errors.  The  group  which  performed 
the  best  received  feedback  in  a  textual  mode  and  had  control  over 
when  and  if  they  were  to  receive  feedback.  The  second  best  group 
received  feedback  in  a  computer  voice  mode  and  had  control  over 
when  and  if  they  were  to  receive  feedback.  The  third  best  group 
received  immediate  feedback  to  errors  and  feedback  that  was  in  a 
textual  mode.  The  group  with  the  poorest  performance  received 
immediate  feedback  to  errors  and  feedback  that  was  in  a  computer 
voice  mode.  The  results  suggest  (1)  that  instruction  in  dynamic 
skill  should  be  non-lntrusive,  and  (2)  that  current  inexpensive 
voice  synthesis  technology  is  not  appropriate  for  dynamic  skill 
training. 


voice  input  in  pynaalc  skill  Training 


Military  team  performance  commonly  requires  the  use  of  voice 
communications  among  team  members.  During  an  air  Intercept  control 
mission,  for  example,  a  team  including  the  Air  Intercept  Controller 
(AIC),  the  Ships  Weapons  Controller  (SWC),  and  the  Tactical  Action 
Officer  (TAO)  must  work  in  a  complementary  fashion,  together  with 
other  members  of  the  shipboard  control  team,  to  advise  the  pilots 
of  Combat  Air  Patrol  (CAP)  aircraft.  Communication  among  these 
team  members  is  largely  conducted  by  voice.  The  armed  forces 
ordinarily  prescribe  the  use  of  voice  brevity  codes  for  team  member 
communications  in  contexts  such  as  air  intercept  control.  The  use 
of  such  codes  helps  to  ensure  the  passing  of  unambiguous  messages. 
It  has  an  additional  advantage  for  the  cause  of  computer  based 
training.  Current  voice  understanding  technologies  require  that 
the  vocal  responses  to  be  received  by  a  computer-based  voice  input 
system  be  part  of  a  finite,  pre- trained  set  of  utterances.  The  use 
of  voice  brevity  codes  in  military  training  thus  makes  those  tasks 
appropriate  for  computer  based  training. 

Voice  has  been  shown  to  provide  a  more  effective  means  of 
communication  between  team  members  working  on  a  common  problem  than 
other  techniques  such  as  writing,  typing,  or  sending  diagrams. 

See,  for  example,  the  work  of  Chapanis  (1975;  Chapanis,  Parrish, 


Ochsnan,  &  Weeks,  1977).  To  attempt  computer-based  team  training 
of  tasks  that  call  for  verbal  communication  by  replacing  that 
coununlcatlon  with  a  different  form  of  comnunicatlon  more  easily 
monitored  by  the  computer  would  be  a  mistake.  Substituting 
keyboard  entry  for  voice  would  make  it  very  easy  for  the 
computer-based  training  system  to  monitor  team  member 
oonaunlcatlons.  Unfortunately,  it  would  also  make  the  team's  task 
very  unnatural  and,  probably,  very  difficult. 

The  use  of  computer  voice  input  in  computer  based  training 
poses  significant  methodological  Issues.  How  can  the  computer 
system  be  reasonably  certain  that  its  perceptions  of  student 
communications  are  reasonably  congruent  with  those  of  the  students? 
That  is,  if  speaker  A  says  something  to  addressee  B,  how  can  one  be 
reasonably  certain  that  what  the  system  hears  is  close  to  what  B 
hears?  One  reason  to  expect  recognition  problems  is  that  even  the 
best  of  the  commercially  available  systems  is  likely  to  make  many 
more  recognition  errors  than  would  take  place  with  keyboard  entry. 

A  much  more  compelling  reason  to  expect  problems,  however,  is  that 
students  will  often  not  restrict  themselves  to  the  required  voice 
brevity  codes  In  speaking.  Since  only  the  pre-established  voice 
codes  can  be  taught  to  the  voice  recognition  system,  innovative 
voice  Inputs  will  result  in  either  Incorrect  recognitions  or 
failures  to  recognize. 


One  response  to  this  dilemma  is  to  avoid  the  use  of  computer- 


based  training  (CBT)  for  team  training  that  requires  voice 
communications.  This  would  be  an  unfortunate  choice,  in  that  it 
would  forsake  the  well-established  advantages  of  CBT,  such  as 
accurate  monitoring  and  record  keeping,  untiring  individual 
attention,  and  reduced  training  personnel  requirements.  To 
Investigate  this  issue,  a  pilot  study  has  been  conducted  to  explore 
the  differences  in  performance  in  single  subjects  on  a  task  using 
either  a  keyboard  or  a  voice  recognition  system.  A  gunnery 
exercise  game  developed  by  Greitzer,  Hershman,  and  Relly(198l, 

Kelly  and  Greitzer,  1982)  and  adapted  to  Apple  Pascal  by  Halff 
(personal  comnunl cation)  was  used,  with  minor  modifications.  In 
the  exercise  it  is  necessary  for  the  student  to  fire  missies  at  an 
optimal  point  in  time  to  receive  the  maximum  points  possible.  If  a 
mlssle  is  sent  too  soon,  the  missle  splashes  into  the  ocean  and 
misses  its  target.  If  it  is  fired  too  late,  points  are  lost.  This 
exercise  should  be  a  sufficient  test  of  the  viability  of  using 
computer  voice  recognition  devices  in  the  analysis  of  military  team 
training  tasks. 

Method 

Subjects.  Subjects  were  paid  volunteers  who  responded  to  posted 
notices  at  the  University  of  Southern  California.  Thirty  students 
participated  in  the  experiment.  All  completed  the  experiment. 
Students  were  assigned  to  one  of  the  two  groups  in  alternating 
sets-of-flve  order  as  they  arrived  for  the  experiment.  Each 
student  received  four  dollars  for  participating  in  the  experiment. 
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Procedure.  Subjects  were  run  Individually  in  the  experiment. 
Completion  of  the  exercise  session  required  from  forty  to  fifty 
minutes.  All  subjects  were  given  an  Information  sheet  which 
explained  the  task.  The  Instructions  were  the  same  except  where  it 
pertained  to  the  sending  of  the  missis  ( i. e . ,  via  keyboard  or 
voice).  In  the  game,  the  student  gunner  views  a  radar  display  in 
which  his  or  her  own  ship  is  in  the  center.  Concentric  rings  mark 
the  twenty-  and  forty-mile  ranges  from  the  ship.  The  radar 
displays  incoming  attacking  aircraft  (bogeys).  The  bearings  of 
these  incoming  bogeys  are  displayed  on  the  radar  screen  as 
two-digit  numbers  near  the  point  of  origin  of  each  bogey  on  the 
radar  screen. 

The  task  of  the  gunnery  student  is  to  observe  the  incoming 
bogeys  and  to  fire  defensive  missiles  at  the  appropriate  time.  The 
apropriate  time  is  determined  by  the  speed  of  the  attacking  bogey. 
If  an  Incoming  attacker  is  moving  very  quickly,  a  countering 
missile  must  be  fired  well  in  advance  of  the  time  it  would  be  fired 
for  a  slower  bogey.  Students  estimate  bogey  speed  based  on  the 
displayed  distance  covered  between  radar  updates. 

In  one  version  of  the  game,  the  student  fires  a  countering 
missile  simply  by  keying  in  the  bearing  of  the  bogey  on  an 
auxiliary  keyboard.  A  ship-to-air  missile  is  then  fired  in  that 
direction.  These  missiles  all  have  the  same  constant  speed  and 


limited  range.  If  the  misaile  is  fired  too  early,  it  will  reach 
the  limit  of  its  range  (20  miles}  and  fall  into  the  ocean  before  it 
encounters  its  target.  If  the  missile  is  fired  late,  the  bogey 
will  be  unacceptably  close  when  it  is  downed.  If  the  missile  is 
not  fired  at  all,  the  bogey  will  hit  the  ship. 

Students  receive  two  points  for  getting  a  bogey  just  at  the 
twenty  mile  range  and  one  point  for  downing  a  bogey  in  less  than 
that  range.  Students  lose  twelve  points  if  their  ship  is  hit. 

Task  difficulty  is  increased  by  increasing  the  number  and  speed  of 
the  attacking  enemy  aircraft. 

In  the  voice  input  version  of  the  game,  the  gunnery  student 
does  not  use  the  keyboard.  Instead,  the  student  simply  says  the 
number,  expressed  as  a  sequence  of  digits,  followed  by  the  word 
"Launch"  to  send  a  ship-to-air  missile  against  a  bogey  at  a 
particular  bearing.  Editing  of  the  input  numbers  is  permitted  by 
either  using  the  word  "Cancel"  which  erases  all  digits  entered  or 
by  entering  more  digits  which  scroll  the  previous  digits  to  the 
left  (i.e. ,  the  right-most  digit  was  the  last  one  spoken). 

Before  a  student  plays  the  Air  Defense  Command  (ADC)  game 
using  the  voice  input  device  (the  Auricle  I),  he  must  first 
"train"  the  Auricle  to  understand  his  spoken  commands.  Since  the 
Auricle  is  limited  to  speaker-specific  recognition,  each  student 
must  train  the  Auricle  to  understand  their  parti cular  voice.  The 
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student  is  prompted  with  each  command  word  and  repeats  this  command 
aloud  three  times.  After  all  twelve  command  words  (Zero  through 
Nine,  Launch,  and  Cancel)  have  been  added  to  the  Auricle's  memory, 
the  student  is  asked  to  repeat  each  command  one  more  time  as  a  last 
check  to  see  if  the  Auricle  will  understand  each  command  during  the 
ADC  game. 

All  subjects  were  guided  by  the  experimenter  through  a 
practice  game  which  consisted  of  eighteen  bogeys.  Next  the 
students  participated  in  two  complete  games  on  their  own.  Each 
game  consisted  of  fifty- four  bogeys.  Data  was  collected 
automatically  by  the  computer.  Data  consisted  of  the  achieved 
score  by  the  students  on  each  game. 

Feaultg 

The  scores  for  the  two  training  practice  games  were  summed  and 
an  analysis  of  variance  was  performed.  See  Table  1.  Although  no 
significant  difference  was  found  between  the  voice  input  and 
keyboard  input  groups,  the  mean  scores  for  the  two  groups  are  quite 
divergent,  89  points  for  the  keyboard  group  and  -24  for  the  voice 
group.  Visual  inspection  of  the  data  suggests  that  a  number  of 
students  in  the  voice  group  performed  poorly  in  the  first  practice 
game  but  improved  in  the  second  one.  A  second  analysis  was 
performed,  using  only  the  scores  from  the  second  game.  See  Table 
2.  The  results  are,  again,  not  significant.  The  keyboard  group 
scored  a  mean  of  54  points,  while  the  voice  group  scored  9  points 


Page  18 


In  this  game.  This  is  an  Improvement  over  the  negative  mean  score 
of  the  voice  group  for  the  combined  games,  but  still  rather 
unimpressive  in  comparison  with  the  keyboard  entry  group. 

Inspection  of  the  performance  data  for  the  second  practice 
game  shows  that  three  of  the  students  in  the  voice  group  had  severe 
performance  difficulties  throughout  the  experiment,  while  the  other 
twelve  had  learned  how  to  use  the  system  effectively  by  the  time  of 
the  second  practice  game.  Those  three  students  earned  negative 
scores  on  the  second  practice  game.  If  their  data  is  removed,  the 
keyboard  and  voice  input  groups  appear  to  have  performed  almost 
identically,  with  the  voice  group  scoring  51  points  on  the  second 
game.  These  results  tentatively  suggest  that  there  is  no 
significant  difference  between  sending  commands  by  /oice  and 
sending  commands  by  keyboard  in  this  task  for  most  of  the  students. 

PAacugaLon 

Results  from  the  pilot  study  indicate  that  available,  moderate 
cost  voice  recognition  systems  may  adequately  replace  keyboard 
technology  for  interactive  training  of  dynamic  skills.  Twenty 
percent  of  the  voice  students  did  have  an  extremely  poor  score, 
which  may  be  due  to  inconsistencies  in  the  pronunciation  of  the 
vocabulary  items.  Those  students  who  spoke  in  a  calm  tone  of  voice 
during  the  template  creation  process,  but  an  excited  tone  of  voice 
during  the  recognition  phase,  suffered  poorer  recognition  results 
than  students  who  spoke  consistently.  It  is  possible  that  this 
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problem  could  be  ameliorated  by  creating  a  more  naturalistic 
setting  for  template  creation. 


In.  Military  Team  learning 


Military  team  training  offers  a  desirable  environment  in  which 
to  study  the  feasibility  of  voice  input  and  output  technologies  for 
computer  based  instruction.  Military  team  training  tasks  are 
usually  well  defined,  making  them  suitable  for  computer  based 
instruction.  Voice  communi cations  in  these  tasks  are  by  means  of 
voice  brevity  codes,  prescribed  utterances  from  a  restricted 
vocabulary.  Because  the  vocabulary  is  restricted,  a  voice 
recognition  system  can  be  trained  in  the  permitted  utterances  for 
each  team  member.  The  use  of  these  codes  by  students  will  be 
encouraged  through  the  use  of  voice-input  training  systems. 

Because  voice  recognition  systems  are  unable  to  recognize  novel 
utterances,  training  systems  that  employ  voice  input  will  encourage 
the  use  of  the  prescribed  brevity  codes. 


The  proposed  method  for  incorporating  computer-monitored  voice 
communications  in  computer-based  team  training  is  to  use  the 
computer  as  an  intelligent  transmission  medium.  Figure  1  sketches 
the  transmission  function  of  such  a  system.  Student  team  members 
are  auditorily  isolated  from  each  other.  A  student  claims  the 


transmission  line  by  depressing  a  switch.  The  computer  system  then 
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loads  into  the  voles  recognition  unit  the  student's  prerecorded 
vocabulary  and  then  signals  the  student  that  the  voice  line  is 
available.  The  student  speaks  in  voice  brevity  code,  which  the 
voice  input  system  recognizes  and  passes  to  the  computer  program. 

If  there  is  a  failure  to  recognize,  the  system  prompts  the  student 
to  repeat.  Vhen  a  recognizable  utterance  is  produced,  the  system 
itself  speaks  this  utterance  to  the  addressee,  using  a  voice  output 
device. 

This  approach  offers  a  number  of  advantages  to  one  in  which  a 
computer  voice  input  device  attempts  to  monitor  the  free  exchange 
of  verbal  communications  between  two  speakers.  One  advantage  is 
that  nothing  can  be  transmitted  from  a  speaker  to  a  listener 
without  the  computer  first  recognizing  the  utterance.  Thus 
communication  is  strictly  controlled.  This  means  that  all  team 
communication  is  accurately  monitored  by  the  system.  A  second 
advantage  is  that  the  use  of  voice  brevity  codes  is  strictly 
enforced.  Students  will  quickly  learn  that  fellow  team  members  can 
only  hear  messages  phrased  in  voice  brevity  codes.  Such  a  training 
system  will  promote  the  acquisition  of  brevity  code  communication 
habits  early  in  training. 

There  are  potential  disadvantages  to  this  training  approach, 
which  should  be  explored  empirically.  The  first  potential  problem 
is  that  the  interposition  of  a  delay  in  voice  transmission,  due  to 
the  system's  recognition  and  relay  times,  may  be  long  enough  to 


restrict  the  effectiveness  of  voice  communication.  Another 
possiblity  Is  thst  the  striot  enforcement  of  the  use  of  voice 
brevity  oodes  may  hurt  performance. 

If  computer-relayed  voice  conmunications  prove  effective 
in  team  training,  several  special  applications  of  the  technique 
deserve  detailed  exploration.  One  such  application  is  individual 
team  member  training  by  the  computer-based  training  system.  If 
team  members  are  isolated  from  each  other  in  the  training 
configuration,  it  may  be  very  easy  for  the  training  program  to 
emulate  certain  members  of  the  team  for  the  purpose  of  individually 
training  other  members  of  the  team.  If  the  emulation  behaves  in 
the  same  manner  as  actual  team  members  in  terms  of  the  transmitted 
voice  codes  and  the  decisions  taken,  student  trainees  should  not  be 
able  to  tell  whether  they  are  interacting  with  the  program  or  with 
other  students. 

A  variety  of  experiments  are  called  for  to  explore  the 
consequences  of  computer-relayed  voice  communications  in  team 
training.  At  least  two  issues  must  be  addressed  by  such 
experiments.  The  first  issue  is  whether  the  use  of 
computer-relayed  voice  causes  significant  deterioration  in 
performance  on  the  task  and  on  learning.  The  second  issue  is 
whether  computer-monitored  voice  can  be  productively  employed  to 
promote  more  effective  training  through  adaptive  system  responses 
to  student  voice  input.  That  is,  can  a  CBT  system  take  advantage 


of  Its  monitoring  of  vocal  interchanges  to  provide  aore  effective 
instruction? 

Programs  have  been  prepared  to  conduct  an  experiaent  to  answer 
this  question.  Teams  of  two  students  work  on  simulated  gunnery 
problems  in  a  context  similar  to  the  Air  Defense  game  used  for  the 
experiment  described  above.  In  the  new  version  of  the  game,  one 
team  member  acts  as  a  radar  operator,  who  observes  the  radar  screen 
and  decides  when  a  missle  should  be  fired  and  at  what  bearing.  The 
radar  operator  passes  these  instructions  to  the  second  student,  who 
plays  the  role  of  the  missle  operator  or  gunner.  The  gunner  fires 
missies  by  keying  in  the  bearing  sent  by  the  radar  operator. 

In  one  condition  of  this  experiment  the  radar  operator  gives 
his  Instructions  directly  to  the  gunner  by  voice.  In  the  other 
condition,  the  radar  operators  voice  instructions  are  intercepted 
by  the  voice  recognition  device,  understood,  and  then  relayed  to 
the  gunner  by  means  of  a  voice  output  device.  In  both  cases  the 
gunner  responds  by  keying  in  the  attack  bearing  and  depressing  the 
"fire"  key. 

The  extent  to  which  the  computer-monitored  voice  transmission 
group  performs  less  well  than  the  direct  voice  communication  group 
will  serve  as  a  measure  of  the  difficulty  imposed  by  the  use  of 
oomputer  voice  I/O  in  this  type  of  training. 


Modeling  the  Acquisition  of  Complex  nvnawip  skills 

Intelligent  training  ays tens  of  the  future  nay  be  ubiquitous. 
Certainly,  they  will  be  ooat  effective,  providing  their  software 
can  be  nade  as  Intelligent  as  we  expect.  One  Important  aspect  of 
providing  intelligent  instruction  is  having  a  reasonable 
understanding  of  the  student.  Research  is  called  for  to  develop  a 
general  theory  of  skill  acquisition  and  a  method  for  the 
representation  of  knowledge  about  dynamic  tasks.  A  training  system 
that  embodies  such  a  theory  should  be  better  able  to  understand  the 
student  than  one  that  does  not. 

Other  benefits  can  be  expected  from  a  coherent  approach  to  the 
representation  of  dynamic  skills.  One  example  is  that  computer 
based  monitoring  of  actual  skill  performance  in  the  field  could  be 
made  possible,  using  realistic  models  of  the  performers'  skills  and 
knowledge  states  relevant  to  the  task.  The  monitor  program  would 
evaluate  the  performance  in  terms  of  a  model  of  the  user  built  up 
in  the  course  of  his  or  her  performance.  Another  possibility  is 
that  when  a  new  dynamic  skill  task  is  designed  (as,  for  example,  in 
the  creation  of  a  new  type  of  vehicular  control  system),  the 
performance  characteristics  of  the  task  could  be  modeled  using  this 
system. 

A  procedural  semantics  format  can  be  employed  for  the 
representation  of  dynamic  skill  knowledge.  This  representation 


format  has  the  advantage  that  it  la  designed  for  implementation  in 
a  computer  simulation  of  a  student's  mental  processes  in  dynamic 
skill  performance. 

Two  alternative  approaches  can  be  developed  to  produce 
descriptions  of  students'  understandings  of  the  components  of  the 
task.  These  descriptions  will  be  used  to  produce  student-specific 
models  of  the  task,  which  will  drive  simulations  of  student 
performance.  A  describe-simulate-verify  cycle  will  be  used  to 
fine-tune  student  descriptions  to  create  accurate  simulations  of 
student  performance.  Repetitions  of  this  process  of  modeling 
lndvldual  student's  acquisitions  of  a  dynamic  skill  throughout  the 
course  of  instruction  will  be  used  to  explore  the  nature  of  skill 
acquisition  in  general.  The  products  of  this  research  will  be  a 
tested  representational  format  for  dynamic  skills  and  related 
knowledge,  a  set  of  methods  for  creating  and  validating  such 
representations,  and  a  model-theoretic  description  of  the 
mechanisms  of  skill  learning.  A  notable  potential  application  for 
these  research  products  is  a  method  for  creating  models  of  the 
student  for  use  in  intelligent  computer-based  dynamic  skill 
training  systems. 

The  most  important  products  of  the  research  will  be  models  of 
the  skills  and  knowledge  required  by  such  complex  tasks  and  a 
theory  of  the  nature  of  the  learning  process  for  these  tasks. 

Major  portions  of  this  theory  will  be  implemented  in  a  computer 
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model  of  the  skills  and  knowledge  required  by  a  complex  dynamic 
skill.  A  theory  of  how  novice  students  develop  and  modify  these 
skill  and  knowledge  representations  will  also  be  developed.  The 
representations  will  be  expressed  as  schemata  within  the  framework 
of  procedural  semantics.  Eventually  this  type  of  representation 
should  play  an  important  role  in  intelligent  computer  based 
instruction  of  dynamic  skills. 
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Analysis  of  Variance 


Group  means  for  first  analysis. 

Keyboard  88.933 
Voice  -24.40 


Source 

Sum  of  Sauares 

41 

Mean  Square 

F  Ratio 

Between 

96333.300 

1 

96333.300 

3.682 

Within 

732559.000 

28 

26162.800 

Total 

628892.000 

29 

Table  1.  Voice  Input  in  Training  Experiment.  Summed  scores 
for  two  training  sets. 


Analysis  of  Variance 


Group  means 

for  second  analysis. 

Keyboard 

54.067 

Voice 

8.867 

Source 

Sum  of  Sauares 

41 

Mean  Square 

F  Ratio 

Between 

15322.800 

1 

15322.800 

2.507 

Within 

171165.000 

28 

6113.020 

Total 

186487.000 

29 

Table  2.  Voice  Input  in  Training  Experiment.  Final  training 
scores. 
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Navy 

1  Robert  Ahlers 
Code  N711 

Human  Factors  Laboratory 
NAVTRAEQUIPCEN 
Orlando,  FL  32813 

1  CDR  Robert  J.  Biersner 
Naval  Medical  R4D  Command 
National  Naval  Medical  Center 
Bethesda ,  MD  20814 

1  Dr.  Alvah  Bittner 

Naval  Biodynamics  Laboratory 
New  Orleans,  LA  70189 

1  Liaison  Scientist 

Office  of  Naval  Research 
Branch  Office,  London 
Box  39 

FPO  New  York,  NY  09510 

1  Chief  of  Naval  Education  and  Training 
Liason  Office 

Air  Force  Human  Resource  Laboratory 
Operations  Training  Division 
WILLIAMS  AFB,  AZ  85224 

1  Dr.  Stanley  Collyer 
Office  of  Naval  Technology 
800  N.  Quincy  Street 
Arlington,  VA  22217 

1  CDR  Mike  Curran 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Code  270 

Arlington,  VA  22217 

1  Dr.  Tom  Duffy 

Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 
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NPRDC 
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1  Dr.  John  Ford 

Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 
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1  Dr.  Jim  Hollan 
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Navy  Personnel  R  4  D  Center 
San  Diego,  CA  92152 

1  Dr.  Ed  Hutchins 

Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 

1  Dr.  Norman  J.  Kerr 

Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  38054 

1  Dr.  Peter  Kincaid 
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Dept,  of  the  Navy 
Orlando,  FL  32313 

1  R.  W.  King 

Director,  Naval  Education  and  Training 
Program 

Naval  Training  Center,  Bldg  90 
Great  Lakes.  IL  60088 

1  Dr.  James  Lester 
ONR  Detachment 
495  Summer  Street 
Boston,  MA  02210 

1  Dr.  William  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command,  Code  00A 
Pensacola,  FL  32508 

1  CAPT  Richard  L.  Martin,  USN 
Commanding  Officer 
USS  Carl  Vinson  (CVN-70) 

FPO  New  York,  NY  0955s 

1  Dr.  George  Moeller 

Director,  Behavioral  Sciences  Dept. 
Naval  Submarine  Medical  Research  Lab 
Naval  Submarine  Base 
Groton,  CT  63409 

1  Dr  William  Montague 
NPRDC  Code  1? 

San  Diego,  CA  92152 


1  LT  Steven  D.  Harris,  MSC,  USN 
RFD  1,  Box  243 
Riner,  VA  24149 
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1  Naval  Ocean  R4D  Agency 
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Attn:  LCDR  J.  D.  McKendrick 
Code  335 

Bay  St.  Louis,  MO  39529 

1  Library.  Code  P201L 

Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 

1  Technical  Director 

Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 

6  Commanding  Officer 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1  Office  of  Naval  Research 
Code  933 

800  N.  Quincy  SStreet 
Arlington,  VA  22217 

6  Personnel  4  Training  Research  Group 
Code  442PT 

Office  of  Naval  Research 
Arlington,  VA  22 217 

1  Psychologist 
ONR  Branch  Office 
1030  East  Green  Street 
Pasadena ,  CA  91101 

1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  4  Studies  Branch 
OP  115 

Washington,  DC  20350 

1  LT  Frank  C.  Petho,  MSC,  USN  (Ph.D) 

.CNET  (N-432) 

NAS 

Pensacola,  FL  32508 


1  Dr.  Worth  Scanland 
CNET  (N-5) 

NAS,  Pensacola,  FL  32508 

1  Dr.  Sam  Schiflett 

Aircrew  Systems  (SY70E) 

Naval  Air  Test  Center 
Patuxent  R:ver,  MD  20670 

1  Dr.  Robert  G.  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987H 

Washington,  DC  20350 

1  Dr.  Alfred  F.  Smode,  Director 

Training  Analysis  4  Evaluation  Group 
Dept,  of  the  Navy 
Orlando,  FL  32813 

1  Dr.  Richard  Sorensen 
Navy  Personnel  R4D  Center 
San  Diego,  CA  92152 

1  Dr.  Frederick  Steinh*»iser 
CNO  -  OP115 
Navy  Annex 
Arlington,  VA  20370 

1  Roger  Weissinger-Baylon 

Department  of  Administrative  Sciences! 
Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Dr.  Douglas  Wetzel 
Code  '2 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Mr  John  H.  Wolfe 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 


1  Dr.  Gary  Poock 

Operations  Research  Department 
Code  55PK 

Naval  Postgraduate  School 
Monterey,  CA  939*10 

1  Dr.  Gil  Ricard 
Code  N711 
NTEC 

Orlando,  FL  32813 
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Marine  Corps 

1  H.  William  Greenup 
Education  Advisor  (E031) 
Education  Center,  MCDEC 
Quantico,  VA  22 134 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington,  VA  22217 

1  DR.  A.L.  SLAFKOSKY 
SCIENTIFIC  ADVISOR  (CODE  RD-1  ) 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 
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1  Technical  Director 
U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  James  Baker 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Beatrice  J.  Farr 

U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 

1  Dr.  Milton  S.  Katz 
Training  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Marshall  Narva 

US  Army  Research  Institute  for  the 
Behavioral  &  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 

Director,  Training  Research  Lab 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Joseph  Psotka 

Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Sasmor 
U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Wisher 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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1  AFHRL/LRS 

Attn:  Susan  Ewing 
WPAFB 

WPAFB,  OH  45433 

1  U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Air  University  Library 
AUL/LSE  76/443 
Maxwell  AFB,  AL  36112 

1  Dr.  Earl  A.  Alluisi 
HO,  AFHRL  ( AFSC) 

Brooks  AFB,  TX  78235 

1  Mr.  Raymond  E.  Christal 
AFHRL/MOE 

Brooks  AFB,  TX  78225 

1  Bryan  Dallman 
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Lowry  AFB,  CO  S0230 
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Bolling  AFB,  DC  20332 

1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 
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AFHRL/OTGT 

Williams  AFB,  AZ  85224 

1  Dr.  Ross  L.  Morgan  (AFHRL/LR) 

WPAFB,  OH  45433 

1  3700  TCHTW/TTGHR 
2Lt  Tallarigo 
Sheppard  AFB,  TX  763 11 

1  Dr.  Joseph  Yasatuke 
AFHRL/OT 

Williams  AFB,  AZ  55224 


Department  of  Defense 

12  Defense  Technical  Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  223^4 
Attn:  TC 

1  Military  Assistant  for  Training  and 
Personnel  Technology 
Office  of  the  Under  Secretary  of  Def 
for  Research  &  Engineering 
Room  3D129.  The  Pentagon 
Washington,  DC  20301 

1  Major  Jack  Thorpe 
DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 


USC/Munro  &  Towne  (NR  154-449) 


8-Nov-82 


PaRe  5 


Civilian  Agencies 


Private  Sector 


1  Dr.  Patricia  A.  Butler 
NIE-BRN  Bldg,  Stop  I  7 
1200  19th  St.,  NW 
Washington,  DC  20208 

1  Dr.  Susan  Chipman 
Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Molnar 

Office  of  Scientific  and  Engineering 
Personnel  and  Education 
National  Science  Foundation 
Washington,  DC  20550 

1  Chief,  Psychological  Reserch  Branch 
U.  S.  Coast  Guard  (G-P-1 /2/TP42 ) 
Washington,  DC  20593 

1  Dr.  Frank  Withrow 

U.  S.  Office  of  Education 
400  Maryland  Ave.  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  &  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 


1  Dr.  Erling  B.  Andersen 
Department  of  Statistics 
Studiestraede  8 
1455  Copenhagen 
DENMARK 

1  Dr.  John  R.  Anderson 
Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr .  John  Annett 

Department  of  Psychology 
University  of  Warwick 
Coventry  CV4  7AJ 
ENGLAND 

1  Dr.  Michael  Atwood 
Bell  Laboratories 
11900  North  Pecos  St. 

Denver,  CO  80234 

1  Psychological  Research  Unit 
Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2800 
AUSTRALIA 

1  Dr.  Alan  Baddeley 
Medical  Research  Council 
Applied  Psychology  Unit 
15  Chaucer  Road 
Cambridge  CB2  2EF 
ENGLAND 

1  Dr.  Patricia  Baggett 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Dr.  Jonathan  Baron 
80  Glenn  Avenue 
Berwyn,  PA  19312 

1  Dr.  Menucha  Birenbaum 
School  of  Education 
Tel  Aviv  University 
Tel  Aviv,  Ramat  Aviv  89978 
Israel 

1  Dr.  John  Black 
Yale  University 
Box  1 1 A ,  Yale  Station 
New  Haven,  CT  08520 
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1  Dr.  Lyle  Bourne 
Deprrtment  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Dr.  John  S.  Brown 
XEROX  Palo  Alto  Research  Center 
3'J32  Coyote  Road 
Palo  Alto,  CA  94?04 

1  Dr.  C.  Victor  Bunderson 
WICAT  Inc. 

University  Plaza,  Suite  10 
1160  So.  State  St. 

Orem,  UT  84057 

1  Dr.  Pat  Carpenter 

Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  John  B.  Carroll 
409  Elliott  Rd. 

Chapel  Hill,  NC  27514 

1  Dr.  William  Chase 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  William  Clancey 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94306 

1  Dr.  Allan  H.  Collins 

Bolt  Beranek  4  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  0213? 

1  Dr.  Lynn  A.  Cooper 
LR  DC 

University  of  Pittsburgh 
393S  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  Kenneth  B.  Cross 
Anacapa  Sciences,  Inc. 

P.0.  Drawer  0 

Santa  Barbara,  CA  93102 


Private  Sector 

1  Dr.  Diane  Damos 
Arizona  State  University 
Department  of  Psychology 
Tempe,  AZ  85287 

I 

1  Dr.  Jeffrey  Elman 
University  of  California,  San  Diego 
Department  of  Linguistics 
La  Jolla,  CA  92093 

1  ERIC  Facility-Acquisitions 
4833  Rugby  Avenue  I 

Be the sd a,  MD  20014 

1  Professor  Reuven  Feuerstein 
HWCRI  Rehov  Karmon  6 
Bet  Hakerem 
Jerusalem 
Israel 

1  Mr.  Wallace  Feurzeig 
Department  of  Educational  Technology 
Bolt  Beranek  A  Newnan 
1C  Moulton  St. 

Cambridge,  MA  02238 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  2055C 

1  Dr.  Dexter  Fletcher 
WICAT  Research  Institute 
1875  S.  State  St. 

Orem,  UT  22333 

1  Dr.  John  R.  Frederiksen 
Bolt  Beranek  &  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Robert  Glaser 
Learning  Research  A  Development  Cente 
University  of  Pittsburgh 
3939  O'Hara  Street 
PITTSBURGH,  PA  15260 

1  Dr.  Marvin  D.  Glock 
217  Stone  Hall 
Cornell  University 
Ithaca,  NY  14853 
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1  Dr.  Daniel  Gopher 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  DR.  JAMES  G.  GREENO 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3929  O'HARA  STREET 
PITTSBURGH,  PA  15213 


Private  Sector 

1  CDR  Robert  S.  Kennedy 
Canyon  Research  Group 
1040  Woodcock  Road 
Suite  227 
Orlando,  FL  32803 

1  Dr.  David  Kieras 
Department  of  Psychology 
University  of  Arizona 
Tuscon,  AZ  85721 


1  Dr.  Harold  Hawkins 
Department  of  Psychology 
University  of  Oregon 

^  Eugene,  OR  97402 

1  Dr.  Barbara  Hayes-Roth 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  95305 

I* 

1  Dr.  Frederick  Hayes-Roth 
Teknowledge 
525  University  Ave. 

Palo  Alto,  CA  94301 

9  1  Glenda  Greenwald,  Ed. 

Human  Intelligence  Newsletter 
P.  0.  Box  1163 

i  Birmingham,  MI  480 12 

1  Dr.  Earl  Hunt 

L  Dept,  of  Psychology 

University  of  Washington 
Seattle,  WA  98105 

1  DR.  KAY  INABA 
21116  VANOWEN  ST 

[•  CAN0GA  PARK,  CA  91303 

1  Dr.  Steven  W.  Keele 

Dept,  of  Psychology 
University  of  Oregon 
Eugene,  OR  97403 

( ' 

1  Dr.  Scott  Kelso 
Haskins  Laboratories,  Inc 
270  Crown  Street 
New  Haven,  CT  06510 
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1  Dr.  Stephen  Kosslyn 
Department  of  Psychology 
Brandeis  University 
Waltham,  MA  02254 

1  Dr.  Pat  Langley 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Marcy  Lansman 

The  L.  L.  Thurstone  Psychometric 
Laboratory 

University  of  North  Carolina 
Davie  Hall  013A 
Chapel  Hill,  NC  27514 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  Lesgold 
Learning  RAD  Center 
University  of  Pittsburgh 
2929  O'Hara  Street 
Pittsburgh,  PA  15260 

1  Dr.  Jim  Levin 
University  of  California 
at  San  Diego 

Laboratory  fof  Comparative 
Human  Cognition  -  DOO’A 
La  Jolla,  CA  92093 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 
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Washington,  DC  20550 

1  Dr.  James  R.  Miller 
Texas  Instruments,  Inc. 

Central  Research  Laboratory 
P.  0.  Box  226015,  MS238 
Dallas,  TX  75266 

1  Dr.  Mark  Miller 

Computer  Thought  Corporation 
1721  West  Plane  Parkway 
Plano,  TX  75075 

1  Dr .  Donald  A  Norman 
Cognitive  Science,  C-015 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Committee  on  Human  Factors 
JH  811 

2101  Constitution  Ave.  NW 
Washington,  DC  20418 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 

Alexandria,  VA  2231 1 

1  Dr.  James  W.  Pellegrino 
University  of  California, 

Santa  Barbara 
Dept,  of  Psychology 
Santa  Barabara,  CA  93106 

1  Dr.  Nancy  Pennington 
University  of  Chicago 
5801  S.  Ellis  Avenue 
Chicago,  IL  60637 

1  Mr.  L.  Petrullo 
2431  N.  Edgewood  Street 
ARLINGTON,  VA  22207 


Private  Sector 

1  Dr.  Richard  A.  Poliak 
Director,  Special  Projects 
Minnesota  Educational  Computing  Conso 
2520  Broadway  Drive 
St.  Paul ,MN 

1  Dr.  Martha  Poison 
Department  of  Psychology 
Campus  Box  346 
University  of  Colorado 
Boulder,  CO  80309 

1  DR.  PETER  POLSON 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
BOULDER.  CO  80309 

1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh.  PA  1521 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
Murray  Hill,  NJ  07974 

1  Dr.  William  B.  Rouse 
Georgia  Institute  of  Technology 
School  of  Industrial  A  Systems 
Engineering 
Atlanta,  GA  ’0332 

1  Dr.  David  Rumelhart 
Center  for  Human  Information  Processin 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 


1  Dr.  Michael  J.  Samet 
Perceptronics,  Inc 
6271  Variel  Avenue 
Woodland  Hills,  CA  91364 
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1  Dr.  Alan  Schoenfeld 
Mathematics  and  Education 
The  University  of  Rochester 
Rochester,  NY  14627 

1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
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300  N.  WASHINGTON  ST. 

ALEXANDRIA,  VA  22314 

1  Dr.  H.  Wallace  Sinaiko 
Program  Director 

Manpower  Research  and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street 
Alexandria,  VA  22314 

1  Dr.  Edward  E.  Smith 

Bolt  Beranek  4  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Richard  Snow 
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Stanford  University 
Stanford,  CA  94  305 

1  Dr.  Eliott  Soloway 
Yale  University 
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Brown  University 
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1  Dr .  Robert  Sternberg 
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Yale  University 
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10  Moulton  St. 

Cambridge,  MA  02238 


1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
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1  Dr.  Kikumi  Tatsuoka 
Computer  Based  Education  Research  Lab 
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